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retrieves application-specific information concerning the 
specified status or operational characteristic of an instance of 
the application. Status objects interact with instances of the 
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LOAD BALANCING IN A NETWORK 
ENVIRONMENT 

U.S. Pat. No. 6,092,178, entitled "Systems for Respond- 
ing to a Resource Request," and U.S. patent application Ser. 
No. 09/146,848, entitled "Load Balancing for Replicated 
Services," both of which were filed on Sep. 3, 1998, are 
related to the present application. 

BACKGROUND 

This invention relates to the field of computer systems. 
More particularly, a system and methods are provided for 
load balancing among application programs or replicated 
services. 

In many computing environments, clients (e.g., computer 
systems and users) connect to servers offering a desired 
application or service — such as electronic mail or Internet 
browsing. One computer server may, however, only be 
capable of efficiently satisfying the needs of a limited 
number of clients. In such a case, an organization may 
employ multiple servers offering the same application or 
service, in which case the client may be connected to any of 
the multiple servers in order to satisfy the client's request. 

A service offered simultaneously on multiple servers is 
often termed "replicated" in recognition of the fact that each 
instance of the service operates in substantially the same 
manner and provides substantially the same functionality as 
the others. The multiple servers may, however, be situated in 
various locations and serve different clients. Application 
programs may also operate simultaneously on multiple 
servers, with each instance of an application operating 
independently of, or in concert with, the others. In order to 
make effective use of an application or replicated service 
offered by multiple servers (e.g., to satisfy clients' requests), 
there must be a method of distributing clients* requests 
among the servers and/or among the instances of the appli- 
cation or service. This process is often known as load 
balancing. Methods of load balancing among instances of a 
replicated service have been developed, but are unsatisfac- 
tory for various reasons. 

In one method of load balancing a replicated service, 
clients' requests are assigned to the servers offering the 
service on a round-robin basis. In other words, client 
requests are routed to the servers in a rotational order. Each 
instance of the replicated service may thus receive substan- 
tially the same number of requests as the other instances. 
Unfortunately, this scheme can be very inefficient. 

Because the servers that offer the replicated service may 
be geographically distributed, a client's request may be 
routed to a relatively distant server, thus increasing the 
transmission time and cost incurred in submitting the request 
and receiving a response. In addition, the processing power 
of the servers may vary widely. One server may, for 
example, be capable of handling a larger number of requests 
or be able to process requests faster than another server. As 
a result, a more powerful server may periodically be idle 
while a slower server is over-burdened. 

In another method of load balancing, specialized hard- 
ware is employed to store information concerning the serv- 
ers hosting instances of a replicated service. In particular, 
according to this method information is stored on a com- 
puter system other than the system that initially receives 
clients' requests. The stored information helps identify the 
server having the smallest load (e.g., fewest client requests). 
Based on that information, a user's request is routed to the 
least-loaded server. In a web-browsing environment, for 
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example, when a user's service access request (e.g., a 
connection request to a particular Uniform Resource Locator 
(URL) or virtual server name) is received by a server 
offering Domain Name Services (DNS), the DNS server 
5 queries or passes the request to the specialized hardware. 
Based on the stored information, the user's request is then 
forwarded to the least -loaded server offering the requested 
service. 

This method is also inefficient because it delays and adds 

10 a level of complexity to satisfying access requests. In 
particular, one purpose of a DNS server is to quickly resolve 
a client's request for a particular service to a specific server 
(e.g., a specific network address) offering an instance of the 
service. Requiring the DNS server to query or access another 

15 server in order to resolve the request is inefficient and delays 
the satisfaction of the request. 

In yet other methods of balancing requests among mul- 
tiple instances of a replicated service, client requests are 
randomly assigned to a server or are assigned to the closest 

20 server. Random assignment of client requests suffers the 
same disadvantages as a round-robin scheme, often causing 
requests to be routed to geographically distant servers and/or 
servers that are more burdened than others. This naturally 
results in unnecessary delay. Simply assigning requests to 

25 the closest server may also be inefficient because a faster 
response may be available from a server that, although 
further from the client, has less of a load. 
As mentioned above, present load balancing techniques 

30 are also limited in scope. For example, the techniques 
described above are designed for replicated services only 
and, in addition, only consider the operational status or 
characteristics of the servers hosting the replicated service, 
not the service itself. In other words, present techniques do 

35 not allow load balancing among instances of an application 
program or, more generally, the collection or consideration 
of information concerning the status of individual instances 
of applications or services executing on multiple servers. 

SUMMARY 

40 

In one embodiment of the invention a system and methods 
are provided for balancing client (e.g., user) requests among 
multiple instances of an application (e.g., application pro- 
gram or replicated service) in accordance with a selected 

45 policy. In this embodiment, each instance of the load- 
balanced application executes on a separate computer server. 

A load balancing policy is selected for distributing the 
client requests among the multiple servers and instances of 
the application and, at periodic intervals, a "preferred" 

50 server is identified in accordance with the policy. 
Illustratively, the selected policy reflects or specifies one or 
more application-specific factors or characteristics to be 
considered in choosing the preferred server. Client requests 
are routed to the preferred server until such time as a 

55 different server is preferred. A selected load balancing policy 
may be replaced while the application continues operating. 
Other exemplary policies reflect preferences for the least - 
loaded instance of the application or the instance having the 
fastest response time. The least-loaded instance may be that 

60 which has the fewest connected clients and/or the fewest 
pending client requests. In another policy, where the closest 
instance of the application is favored, the preferred server 
may be the server that can be reached in the fewest network 
hops or connections. Another illustrative policy favors the 

65 server and/or the instance with the greatest throughput (e.g., 
the highest number of client requests satisfied in a given time 
period). 
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Depending upon the selected policy, status objects (e.g., FIG. 4 is a flow chart demonstrating the generation of 

agents, modules or other series of executable instructions) objects in a load -balancing framework in accordance with an 

are configured to collect these various pieces of information embodiment of the present invention, 

from each instance of the application that is being load- nG 5 is a flow charl demonstrating the registration of 

balanced (and/or its server). Status objects m one embodi- 5 ob . te vvilhin a load balancing f ram ework and their use in 

ment of the invention thus retrieve application -specific monitori M inslarjce of a loa d-balanced application in 

information (e.g., number and/or type of pending chen accordance ^ M embodiment of the present invention, 

requests) and/or information concerning a server s general r 

status (e.g., its distance from another network entity). DETAILED DESCRIPTION 
Illustratively, each instance of a load-balanced application is 

associated with its own status object(s). In one embodiment The following description is presented to enable any 

of the invention multiple status objects having different person skilled in the art to make and use the invention, and 

functions are associated with one instance. is provided in the context of particular applications of the 

Each instance of the application (or, alternatively, each invention and their requirements. Various modifications to 

server hosting an instance of the application) is also asso- the disclosed embodiments will be readily apparent to those 

ciated with an individual monitor object or IMO (e.g., 15 skilled in the art and the general principles defined herein 

another object, module or series of executable instructions). may be applied to other embodiments and applications 

Each IMO invokes and stores information from one or more without departing from the spirit and scope of the present 

status object(s) collecting information concerning an invention. Thus, the present invention is not intended to be 

instance of the application. In one embodiment of the limited to the embodiments shown, but is to be accorded the 

invention each IMO is configured to interact with a single 20 ^est scope consistent with the principles and features 

status object; in an alternative embodiment multiple status disclosed herein 

objects are associated with an IMO. In addition, in one Inparlicu ia r , illustrative embodiments of the invention are 

embodiment of described in the context of applications such as a database 

with its status obiect(s); in another embodiment each status mD ..^ . , . .. „. „ . 

object stores its application-specific information for retrieval 25 management system (DBMS), electronic mail, or web 

bv the IMO browsing. Various embodiments of the mvention may there- 

A replicated monitor object (RMO) or module is ^ in ™ lve the ^ TC f a central se ™ r ' SUCh * 

employed to collect information from the IMOs associated Name Services (DNS) server, to resolve an access request 

with the various instances of the load-balanced application. for an application into an address of a physical machine such 

The RMO stores this information, which is then analyzed to 30 as a computer server. One skilled in the art will recognize 

identify a preferred server in accordance with the selected that the present invention is not limited to the applications 

policy described herein or the use of a DNS server, and may be 

In an embodiment of the invention in which clients access readily adapted to other applications and services for which 

the application through a central server such as a Domain load balancing is appropriate. 

Name Services (DNS) server, a specialized updater object 35 Illustratively, the program environment in which a present 

updates a lookup table (e.g., a DNS zone file) to identify the embodiment of the invention is executed incorporates a 

preferred server (e.g., by its network address or an alias). In general-purpose computer or a special purpose device such 

this embodiment the lookup table is used to resolve a virtual a hand-held computer. Details of such devices (e.g., 

server name (e.g., a virtual identity of the application) to a processor, memory, data storage and display) are well 

particular server offering an instance of the application. knQwn and m oimUe d for the sake of clarity. 

When a client requests an applicati ™ a ^ual , i a me the « understood that the techniques of the 

central server directs the request to the server indicated in . . , „•„ , rtf 

the lookup table (i.e., the preferred server). The specialized P«" mi S ht Z ? a 8 ZTu* n 

object is thus configured to update the lookup table (or other technologies For example, the methods described herein 

data structure) or otherwise cause the direction or may be implemented in software running on a computer 

re-direction of load-balanced requests to the preferred 45 system, or implemented in hardware utiUzing either a com- 

server bination of microprocessors or other specially designed 

In one embodiment of the invention the status object(s) application specific integrated circuits, programmable logic 

and an IMO execute on each individual server hosting an devices, or various combinations thereof. In particular, the 

instance of the load-balanced application, while the RMO methods described herein may be implemented by a series of 

and updater objects operate on a central server. In an 50 computer-executable instructions residing on a storage 

alternative embodiment, only the status object(s) execute on medium such as a earner wave, disk drive, or computer- 

the individual servers with the application instances. The readable medium. In addition, although specific embodi- 

other objects are distributed among the central server and ments of the invention are described using object-oriented 

other intermediate servers. software programming concepts, the mvention is not so 

55 limited and is easily adapted to employ other forms of 

DESCRIPTION OF THE FIGURES directing the operation of a computer. 

FIG. 1 is a block diagram depicting an illustrative envi- i D a present embodiment of the invention, information 
ronment in which an embodiment of the present invention concerning instances of an application (e.g., an application 
may be implemented to load balance client requests among program or replicated service) operating on multiple corn- 
multiple instances of an application, 50 puter servers is collected and analyzed to identify a "pre- 

FIG. 2 is a block diagram depicting a method of balancing ferred" server. Illustratively, a preferred server is the server 

client requests among application instances in accordance to which client requests for the application are to be routed 

with an embodiment of the present invention. for processing. A preferred server is identified on a regular 

FIG. 3 is a block diagram depicting a method of balancing or periodic basis, and may be the same as or different from 

client requests among geographically dispersed application 65 the server previously identified. By periodically changing 

instances in accordance with an embodiment of the present the preferred server, client requests are load-balanced 

invention. between the participating servers. Individual clients may 
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thus be routed to, and their requests (e.g., database access, implemented to balance client requests among multiple 

send electronic mail, browse a web page) satisfied by, any of instances of an application executing on multiple servers. In 

the multiple servers. the embodiment central server 100 is a computer system that 

The information that may be collected concerning an receives information from the various application instances 

instance of the program illustratively includes its response 5 (and possibly the servers hosting the application instances) 

time for a client request, its operational status (e.g., whether and routes requests from clients such as client 120 to a 

it is up or down), the number of clients connected to the preferred server. In one embodiment of the invention, central 

instance, the number of client requests pending with the server 100 is a DNS server. Back-end or host servers 110, 

instance, its throughput (e.g., number of client requests 112 and 114 each offer one or more instances of application 

handled in a period of time), etc. Information concerning the 10 104, represented by the numerals 104a, 1046 and 104c. 

status or performance of the host servers themselves (e.g., Servers 110, 112 and 114 may be geographically or logically 

load, capacity, distance from a central server) may also be separated from one another. 

collected and analyzed as part of the process of choosing a Central server 100 includes lookup table 102 for resolving 

preferred server. requests for application program 104 to an address of a 

Illustratively, a central server that distributes client 1S server offering an instance of the program. Lookup table 102 

requests for the application among the various instances uses thus includes an entry for the program's identity as exposed 

a lookup table or other data structure or means to store an to clients (e.g., an alias or a virtual server name), to allow the 

identifier of the current preferred server. The central server clients to access an instance of the application on server 110, 

is, in one embodiment of the invention, a Domain Name server 112 or server 114. Thus, the lookup table entry for 

Services (DNS) server. In this embodiment, the application 20 application 104 may indicate a network address (e.g., an IP 

is exposed (e.g., identified) as a virtual server name to which or Internet protocol address) for one of servers 110, 112 and 

clients connect and which the DNS resolves to an address of 114. 

one of the multiple servers operating an instance of the Id the embodiment client 120 is illustratively a personal 

application. computer or workstation configured to provide a user access 

The specific information that is collected (from the van- 25 to a network (e.g., the Internet) and various applications and 

ous application instances and, possibly, the host servers) is services on servers 110, 112 and 114. Client 120 is thus 

determined by a load balancing policy that may be selected coupled to central server 100 via network 122, and includes 

by a system manager or administrator. The preferred server instructions (e.g., a web browser) for communicating via 

is then selected by analyzing the collected information. network 122. Client 120 further includes common compo- 

Thus, in one illustrative policy, the preferred server is the 30 nents such as a processor, memory, storage, input and output 

server offering the application instance that is least-loaded devices, etc. Such common components are well known to 

(e.g., has the fewest pending client requests or fewest those skilled in the art and are omitted from FIG. 1 for the 

connected clients). In another illustrative policy, the pre- purpose of clarity. 

ferred server is the server closest to the central server. ^ In the environment of FIG. 1, when client 120 attempts to 

In this embodiment the various pieces of information are connect to application 104, the access request is received by 

collected and assembled on the central server. After a central server 100. Central server 100, through lookup table 

preferred server is identified, the central server's lookup 102, identifies a preferred server offering an instance of 

table is updated with an identifier (e.g., a network address) program 104 and routes the client request accordingly. The 

of the preferred server and subsequent requests for the 4Q server identified in lookup table 102 may be determined 

application or replicated service are directed to that server. according to a load-balancing policy, as discussed below. 

For example, in a web-browsing environment a DNS zone Further, the server identified in lookup table 102 is updated 

file is updated to indicate that requests for the Internet or changed from lime to time in accordance with the selected 

service or web page are to be routed to the preferred server. policy in order to distribute client requests among the 

In one embodiment of the invention a standard application 45 instances of the application, 

programming interface (API) is provided to construct and In a present embodiment of the invention, information 

apply the load balancing framework described below. With reflecting the status or operation of application instances 

the standard API, a programmer may generate application- 104a, 104/) and 104c (and/or servers 110, 112 and 114) is 

specific status objects (described in detail below in conjunc- collected and analyzed on a regular or periodic basis. The 

tion with FIG. 2) which, when executed, gather the infor- 50 information that is collected is identified in a load balancing 

mation described above. The application-specific status policy that identifies one or more factors or pieces of 

objects may, in addition, interact with the application in information to be used to identify a "preferred" server to 

accordance with an application -specific API. which client requests for application 104 are to be routed. 

Generating application-specific status objects or modules Different policies thus require different information to be 

illustratively allows the collection of any information that 55 c <> llected from the application instances, and the active 

could form the basis for load balancing client requests. For P^icy can be changed during load balancing, 

example, to load-balance a database application, it may be The various pieces of information that may be collected 

desirable to determine the number of users being serviced by illustratively include data such as: whether a server or 

each instance of the application, the number of users that instance of application 104 is operational; the response time 

have accessed an instance, or the number of access requests 60 for a request submitted to a server or application instance; 

that are pending with or that have been processed by each the number of requests processed by or pending on a server 

instance. The information gathered by the application- or application instance, a server's proximity to the central 

specific status objects is used by other objects and/or mod- server (e.g., the number of network hops necessary to reach 

ules in the load-balancing framework in order to determine the server), etc. 

a preferred server. 65 In one embodiment of the invention, status objects are 

FIG. 1 is a block diagram depicting an illustrative envi- generated or produced to collect application-specific data 

ronment in which an embodiment of the invention may be from the application instances. The status objects may be 
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constructed according to a standard API for a present In addition, status objects are configured to retrieve their 

load-balancing framework. Status objects and the load- specified information on a periodic basis. If a status object 

balancing framework are described in detail below with fails to gather its information, it may be assumed that the 

reference to FIG. 2. In one particular embodiment, status associated application instance is not operational. If an 

objects (and other objects within the load-balancing s application instance may be to be down, the associated 

framework) are designed (e.g., an object class is server is removed from consideration as the preferred server 

constructed) according to the standard API in a generation f or mat application 

stage Tien, in a registration stage, individual Illustratively, status objects 200, 202 and 204 communi- 

instantiated from the clashes). Finally, in a monitoring Jy i nA 

stage, the objects begin collecting information. 1(J «te with or access application ins ances 104*, 1046 and 

In the illustrated embodiment of the invention, status 30 ^ m accordance with an application-specific API. Each 

objects periodically interact with instances of application stan * object also illustratively performs a single function or 

104 to collect application-specific statistics that will be used retrieves a slD S le of apphcahon-specific information, 

to select a preferred server. For example, if application 104 In alternative embodiments of the invention however, a 

were a DBMS, a status object may gather the number of „ ™& status object may perform multiple functions or 

database accesses, the number of requests received or 15 P, roduce : multl P L le P ieces of information. For example, in one 

pending, etc. for one instance of the application. As another alternative embodiment, a status object may retrieve mul- 

example, if application 104 were an electronic mail ^ P l f es ° f , ^formation concerning an application 

program, a status object may periodically gather the number ^stance's load (e.g. number of connected clients number 

of inbound and/or outbound messages in queue, the number m of pending requests). The multiple pieces of information 

and size of mailboxes, etc. 20 . lhe ° te combined, (e*. ™ a specified formula or 

Besides status objects, other computer-readable ins true- 10 P^ uce a sin S le value or representation of the 

tions (e.g., in the form of objects, agents or modules) are also ins ance s oa . ,«« V ,„ A ™ j 

executed (also described below) to collect, assemble and ^ FIG. 2, individual monitor objects (IMO) 210, 212 and 

analyze the various pieces of information provided by the 214 reside and execute on servers 110, 112 and 114. Indi- 

status objects and to update lookup table 102. The objects or vidual monitor objects are known as server monitor objects 

agents within a load balancing framework may be created in in one embodiment of the invention. A separate IMO is 

a suitable programming or script language and then config- depicted for each application instance. In particular, IMOs 

ured and installed on each of servers 110, 112 and 114 and/or 210, 212 and 214 collect information from status objects 

on central server 100. 30 200, 202 and 204 respectively. 

In an alternative embodiment of the invention, instead of In one embodiment of the invention, a status object 

returning an address of a server in response to a request for collects the specified application-specific information and 

application 104, the lookup table returns an identifier (e.g., stores it on its host server for collection by the associated 

file name) of a set of instructions. The instructions are IMO. In another embodiment of the invention, status objects 

executed, illustratively by central server 100, in order to 35 interface with and directly communicate the information to 

perform a variety of actions (e.g., load or mount an alternate their associated IMOs. 

Internet or domain namespace). In the embodiment illustrated in FIG. 2, different types of 

FIG. 2 depicts an illustrative embodiment of the invention status objects are executed or invoked with differing degrees 

in which operational and statistical information is collected of regularity. When the status objects collect the application 

from application instances 104a, 104/? and 104c on servers 40 instances' response times, for example, status object 200 

110, 112 and 114, respectively. The collected information is may execute relatively frequently (e.g., every 60 seconds), 

analyzed on central server 100 to choose a preferred server, In contrast, when the status objects reflect a policy preferring 

and lookup table 102 is then modified to reflect an identity the closest server, status object 202 may execute only 

(e.g., a network address) of the preferred server. occasionally (e.g., once per day) because the distance from 

In the illustrated embodiment, application instances 104a, 45 cenlral server 100 t0 m * unlikel Y t0 cfa ange very 

1046 and 104c include application -specific information that often. 

is to be considered in choosing the preferred server. Status Although each IMO is associated with only one status 
objects 200, 202 and 204 therefore execute on servers 110, object and one application instance in the illustrated 
112 and 114, respectively, to gather the information or embodiment, in an alternative embodiment of the invention 
statistics from their associated application instances. The 50 an IMO may collect data from multiple status objects. In this 
status objects advantageously adhere to the format provided alternative embodiment, for example, an IMO may interface 
by a standard API, concerning the manner in which the with one status object to determine the response time of an 
information is to be communicated to the central server. In application instance or server and another status object to 
particular, the status objects are designed to accumulate, determine the load on the instance or server, 
store and/or provide application-specific data for retrieval by 55 Replicated monitor object (RMO) 220 retrieves the col- 
individual monitor objects 210, 212 and 214, which also lected information produced from each IMO associated with 
execute on servers 110, 112 and 114, respectively. an application. Therefore, in the illustrated embodiment 
The configuration of the status objects (e.g., the data they where each of servers 110, 112 and 114 operate a separate 
collect) depends upon the policy that has been selected for instance of a load-balanced application, RMO 220 collects 
choosing a preferred server. For example, where the selected 60 data from IMOs 210, 212 and 214. If the servers also offered 
policy requires choosing the least-loaded server (e.g., the another load -balanced application or replicated service, a 
server having the least-loaded instance of the application), a second RMO may operate on central server 100 for the 
status object may be configured to retrieve the number of purpose of retrieving information concerning that applica- 
pending client requests or number of connected clients. As tion from a different set of IMOs. A replicated monitor 
another example, status objects 200, 202 and 204 may be 65 object may also be known as a central monitor object due to 
configured to retrieve a response time or throughput of their its coordination role on behalf of a central server that 
associated application instances. receives multiple requests for an application. 
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Various means of communication may be employed operates similarly to the RMO described above in conjunc- 

between an RMO and the IMOs associated with a particular tion with to FIG. 2, but in this embodiment is located on a 

application. In a present embodiment of the invention Object server situated between central server 100 and the servers 

Request Broker (ORB) technology is employed. In an alter- offering the application. The load balancing framework of 

native embodiment of the invention Remote Procedure Call 5 the illustrated embodiment also includes status objects (e.g., 

(RPC) technology can be used. depicted by numerals 302a, 304a, 312a and 314a) and IMOs 

In summary, when load balancing is performed in accor- (e.g., depicted by numerals 3026, 3046, 3126 and 3146) 

dance with the embodiments of the invention described operating on servers 302, 304, 312 and 314. 

above, a status object gathers load and/or operational infor- RMO 320 operates on central server 100 to collect data 

mation for an instance of the application being load- 10 from the IRMOs within each server farm. Updater object 

balanced. An IMO interfaces with or otherwise retrieves the 322 updates lookup table 102 to reflect the preferred server 

information from each status object and an RMO gathers the identified from the data collected by RMO 320. 

information from all application instances from the IMOs. In an alternative embodiment of the invention in which an 

The data collected by RMO 220 from the various IMOs application is offered on multiple servers, one or more of 
is analyzed in accordance with the selected policy and a 15 which are local and one or mo re of which are remote, aspects 
preferred server is identified. Illustratively, updater object of the embodiments of the invention depicted in FIGS. 2 and 
230 performs the analysis and selection of a preferred server. 3 are combined. In this alternative embodiment, intermedi- 
As discussed above, the preferred server may, for example, ate servers with IRMOs are employed in server farms 
be the one having the application instance with the fastest comprising the remote servers, in order to pass data between 
response time, the fewest pending client requests, the great- 20 the remote servers' IMOs and an RMO, as in the embodi- 
est capacity for client requests, etc. Illustratively, RMO 220 ment depicted in FIG. 3. Local servers, however, employ 
maintains a data structure (e.g., array, vector, table, IMOs that communicate with the RMO without an inter- 
database) identifying each application instance and/or server vening IRMO, as in FIG. 2. 

that is being load-balanced, along with one or more values In another alternative embodiment of the invention, load 

or other indicators or summaries of the collected information 25 balancing among instances of an application is performed 

concerning each application instance. among multiple participating servers wherein one or more of 

Finally, updater object 230 updates lookup table 102 after the servers are segregated (e.g., situated in a remote location 

the collected information is analyzed and a preferred server and/or within a server farm). Within the group of segregated 

is selected. Illustratively, one updater object is used to servers, a "local" load balancing policy may be implemented 

update the lookup table for all applications being load- 30 for distributing all client requests sent to the group and/or to 

balanced. However, in an alternative embodiment of the a specific member of the group. In this alternative 

invention separate updater objects may be employed for embodiment, the segregated servers may be considered a 

each application. single entity for the purposes of a "global" load balancing 

In the embodiment of the invention depicted in FIG. 2, 35 policy specifying the manner in which client requests for the 

RMO 220 retrieves the collected data and updater object 230 application are to be distributed among participating servers, 

updates the lookup table on a periodic basis. The identity of The global and local policies need not be equivalent (e.g., 

the preferred server may thus change over time so that the the global policy may require selection of the closest server 

client requests are distributed among all active application (or group of servers) while the local policy may require the 

instances. least -loaded server or application instance). 

The status objects, IMOs, RMO and updater object may With reference now to FIGS. 4 and 5, an illustrative 

be considered to comprise a load-balancing framework for method of load balancing between multiple instances of an 

distributing client requests among various instances of an application is depicted. In the illustrated method, a central 

application. As one skilled in the art will recognize, the server (e.g., a DNS Server) resolves client requests for a 

different objects within the framework may be distributed 45 virtual name by which the application is known into an 

among the servers hosting application instances, a central identifier of a preferred server offering an instance of the 

server, and other entities such as intermediate servers. application. Each instance of the application illustratively 

FIG. 3 depicts an alternative embodiment of the invention °P«tes on a and * modifi u ed t0 P u roduce 

in which servers offering an application are geographically application-specific information needed to choose the pre- 

dispersed. In FIG. 3, server farm 300 represents a first 50 ferred server 

collection of servers offering the application (e.g., applica- FIG. 4 demonstrates an illustrative generation stage of the 

tion instances 104j and 1046) and server farm 310 repre- method, in which objects in the load-balancing framework 

sents a second collection of servers offering the same are designed (e.g., object classes are constructed). FIG. 5 

application (e.g., application instances 104c and 104d). demonstrates illustrative registration and monitoring stages, 

Although server farms are depicted in FIG. 3 with multiple 55 in which individual objects are created (e.g., instantiated) 

members (i.e., servers 302 and 304 in server farm 300 and and begin collecting information from instances of the 

servers 312 and 314 in server farm 310), a server farm may load-balanced application. 

consist of any number of members, even one. With reference now to FIG. 4, state 400 is a start state. In 

Each server farm in the presently described embodiment state 402 a policy to be applied to identify a preferred server 

includes an intermediate server (i.e., server 306 in server 60 is selected. One skilled in the art will appreciate that various 

farm 300 and server 316 in server farm 310). One function policies are possible, depending upon the nature of the 

of an intermediate server in this embodiment is to collect, application and the aspect(s) of the application that are 

from the servers in the farm that host instances of the conducive to load balancing. 

load-balanced application information necessary to select a Illustrative policies in a present embodiment of the inven- 

preferred server. For example, intermediate replicated moni- 65 tion focus upon the status or availability of the various 

tor object (IRMO) 306a is operated on intermediate server instances of the application. Such policies reflect prefer- 

306 to collect data from servers 302 and 304. IRMO 306a ences for the least loaded instance, the instance with the 
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fastest response time or throughput, the instance with the In state 410, the existing load balancing framework is 

fewest connected clients, etc. For example, where access examined to determine whether an RMO already exists for 

requests for a database management system (DBMS) are receiving data from the IMOs that are associated with each 

load balanced, illustrative policies may include routing instance of the application. As described above, in one 

requests to the server on which the fewest DBMS requests 5 embodiment of the invention an RMO comprises a data 

have been processed or the server having the fewest con- structure for retaining application-specific information from 

nected users or the fewest unfulfilled processing or access lne application instances. If an RMO already exists, the 

requests. For each application for which requests are load- illustrated method continues at state 414. Otherwise, in state 

balanced, separate policies may be employed. 4U aQ RMQ struclure (egtj an class ) ^ constructed 

In an alternative embodiment of the invention, policies 1Q lhat fc specific l0 tDe application. As with the status objects 

require examination of the availability or status of the and IMOs> aQ actuaJ RMQ ms{aQCC wiu ^ created as part 

servers offering instances of the application. Such policies Qf mc registralion stage depicted io nG> 5 , 

may express preferences for the server having the shortest J .^ J . L .. , JL , * r i ■ 

distance to the central server, the fastest response time, the In sta ' e 414 > lhe ex *» n S load balancing framework is 

best throughput, etc. c examined 0Qce more - ™* tune > « 15 determined if the 

. , • j * . a . . * A 15 sequence of instructions or executable code for the special - 

In general, the selected policy reflects whichever aspect or r^. ,. t . t ... , , . f . r . . 

♦ r »u i j u i a v f«™ tu a uLi* f n * ized object that will determine a preferred server already 

aspects of the load -balanced application lorm the basis tor . . t . r , ,. # , ^ , J 

* ..... e exists. If not, instate 416 a specialized object structure (e.g., 

distributing client requests among the various instances of ' . . \ . . J , . .\ 

& . jj .i_ i_ .* .u r .■ an obiect class) is constructed to apply the selected load 
the application and/or the servers hosting the application J . . ' . . - . ^ r / „ . am 
KF „ . r . n .i_ ~f * * balancing policy to the results of the data collected concern- 
instances. The information reflecting these aspects is pen- . iL . ■ . / Al .u • u . 

.... . r , . 4 i_ . ^ u- * \- 20 mg the various application instances (and/or their host 

odically captured for each instance by status objects working 6 \ j i . * a -ru ■ r ^ u* . 

. ! J ^ %u .u i- '^t««™ servers) and select a preferred server. The speciahzed object 

in close cooperation with the application instances^ fa to £ ^ ^ ^ ^ 

In state 404 sequences of instructions or executable code %Mx ^ {Q stofe ^ ^ Qf ^ ferred 

are produced for performing the function(s) of the status ^ . r L ■.. ^ i_ JL j 

objects (i.c. f to collect the application -specific information 25 ™ e generation stage of the illustrated method then ends 

needed to choose a preferred server). In one embodiment of Wlth end state 418 ' 

the invention in which the load balancing framework is With reference now to FIG. 5, illustrative registration and 

constructed using an object-oriented programming monitoring stages of the illustrated method are depicted. For 

language, a compatible language and basic building blocks present purposes, the term registration refers to the regis- 

provided by the framework are used to generate the status 30 Nation of individual objects (e.g., status object, IMO, RMO, 

objects, IMOs, RMO and specialized object. Thus, in this specialized object) within a load balancing framework, 

embodiment of the invention state 404 comprises the ere- including their creation (e.g., instantiation) from the object 

ation of one or more classes of status objects, from which structures (e.g., classes) produced in the generation stage 

individual instances will be created in the registration stage depicted in FIG. 4. In the monitoring stage, information is 

depicted in FIG. 5. Illustratively, status objects are substan- 35 collected for the purpose of identifying a preferred server in 

tially similar for each instance of a load-balanced applica- accordance with a selected load balancing policy. In FIG. 5, 

t i on state 500 is a start state. 

Status objects may be configured to store the information In state 502, a status object is registered with the load- 
for retrieval by individual monitor objects or, alternatively, balancing framework. In one embodiment of the invention, 
to interface with the IMOs directly in order to pass the 40 the standard API provided with the load balancing frame- 
information along. In addition, the status objects may be work includes a command (e.g., "create") for creating an 
configured to execute automatically on a regular basis, in instance of each object within the framework. As one skilled 
response to action by another part of the load balancing in the art will appreciate, creating an instance of an object, 
framework (e.g., upon invocation by an IMO), the applica- such as a status object, involves the dynamic loading and 
tion or some other external entity, etc. 45 executing of a sequence of instructions defining the object. 

As discussed above, in a current embodiment of the In state 504, configurable parameters of the status object 
invention status objects (and other framework objects) are are set in accordance with the selected policy. Illustrative 
constructed using an object-oriented programming lan- parameters include the frequency with which the 
guage. One skilled in the art will recognize that many application-specific information should be gathered, a net- 
suitable programming languages and tools exist and that the 50 work or port address for communicating with the application 
invention may be implemented using techniques other than instance, information detailing how to communicate with 
object-oriented programming. Illustratively, however, status the application instance and/or IMO, etc. One skilled in the 
objects substantially adhere to a common format (e.g., art will appreciate that a status object may have a variety of 
detailed in a load balancing framework API) in order to configurable parameters, depending upon the nature of the 
cooperate with the overall load balancing framework. 55 application and the selected pohcy. 

In state 406, the existing load -balancing framework is In state 506, an individual monitor object (IMO) is 

examined to determine whether an IMO (e.g., an IMO class) registered with the load-balancing framework. Illustratively, 
already exists for collecting data concerning an instance of one IMO is registered or created for each instance of the 

the load-balanced application. If an IMO already exists, the application. Each IMO may be installed on the server 

illustrated method continues at state 410. Otherwise, in state 60 executing the associated instance of the application. In an 

408 an IMO structure (e.g., an object class) is constructed alternative embodiment, however, IMOs operate on the 

that is specific to the application instance. The IMO is central server or an intermediate server located between the 

designed such that it will collect the various data and central server and the host servers. As described above, 

statistics gathered by one or more status objects). In an IMOs may be configured to collect and report certain 

alternative embodiment of the invention, the IMOs gener- 65 information or data. In the presently described embodiment 

ated for all instances of a particular application are substan- of the invention, the collected information is received 

tially similar. directly from a status object. In an alternate embodiment of 
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the invention, the information may be retrieved from a After the various executable objects or program modules 

location in which it was placed by the status object (e.g., a are configured and installed in states 502-516, the collection 

storage device, a file or other data structure). of server/application information can begin. Thus, in state 

As described above, the information to be collected may 518 the created objects (e.g., status objects, IMOs, RMO and 

be determined by the selected load balancing policy, and will 5 specialized object) are activated or executed if they are not 

be used to identify a preferred server. In a present embodi- already executing. 

ment of the invention, the active policy for an application In state 520) a status object begins collecting or gathering 

may be changed without disrupting ^the handling of client mformation from ils applicalion instance. For example, 

requests. IU^tratively, th* is done by temporarily pausing where fc ^ u favQrs ^ leasMoaded applicatioa 

the operation of IMOs for the apphcation, installing new 3Q slatus ^ obje ct retrieves data concerning an 

status objects reflecting the new policy, then resuming the . 4 . , / J . f & , 

IMOs. Advantageously, the IMOs need not be altered or "J? 130 ? s load («•&. -""nberof client requests or connected 

replaced. cheQls )- 

In state 508, parameters are set for the IMO created in In state 522 80 IM0 retrieves ^ information gathered by 

state 506. Illustrative parameters include a list of status « lls associated status object(s). Then, in state 524, an RMO 

objects from which to collect information, the frequency 15 calls ' mvokes or otherwise communicates with the IMO to 

with which to collect the information, how to communicate retrieve information. The RMO may similarly commu- 

with the status objects and/or RMO, etc. nicate mth additional IMOs storing information concerning 

In state 510 a replicated monitor object is created for the *^ ere 0r instances ° f ,he »PPli«lioo. IUustratively, 

load balanced applicalion. As described above, Ibe RMO 20 ! he . 9 e ™ ules on r lhe ™ l !? A J*™ T and sl °T £ e 

„ . • . ,,7 t . A ^ m ™, rt .^t^o information retrieved from the IMOs for analysis by the 

may be installed on the central server and communicates . J J 

with the IMOs using a suitable format or protocol (e.g., ORB specialized object. 

or RPC). In an alternative embodiment in which intermedi- In slate 526 the ""formation collected by the RMO is 
ate servers are employed (e.g., where remote servers or analyzed in accordance with the selected policy to choose a 
server farms are included), an intermediate RMO is created „ preferred server. In state 528 the specialized object updates 
for each intermediate server. In state 512, RMO parameters the looku P tabIe for tDe « ot "l t0 * dica te the 
are set, possibly including a list of IMOs, the frequency with preferred server. Illustratively, the update procedure com- 
which data is to be collected from the IMOs, a method of P rises associating an alias or network address of the pre- 
communicating with the IMOs, etc. ferred server witb tne name of a virtual server/service 
A backed or host server (e.g., server 110 from FIG. 1) 30 thr ° Ugh whicb c J ienls ac ff the Motion. In addition, in 
may be removed from or added to the load-balancing a .P re f nt embodune nl ; of Jthe invention the central server is 
scheme without significantly disrupting operation of the sl S naled t0 reload the looku P tablc ' Slale 530 15 a f n ead slate ' 
application. A host server may, for example, become inop- ™ e foregoing descriptions of embodiments of the wven- 
erative or require replacement. Illustratively, each RMO < ion have been Presented for purposes of illustration and 
maintains an index (e.g., in an array, linked list, vector, other 35 descri P tion ^ ^ are n0 [ inlended , to be , exhaustive or 
data structure, etc.) of all servers participating in the load l 5> hmit the invention to the forms disclosed. Many modi- 
balancing (e.g., all servers offering an instance of the ficaUons ™ 6 variations will be apparent to practitioners 
application). This information may, for example, be included skilled in the art - Accordingly, the above disclosure is not 
in a list of IMOs from which the RMO receives information. intended to limit the invention; the scope of the invention is 
By temporarily pausing the RMO, removing the IMO asso- 40 dcfined b V thc a PP™ded claims. 

dated with the server from the list and restarting the RMO, In one alternative embodiment of the invention, for 
the RMO will stop attempting to retrieve information for the example, clients access an instance of the application pro- 
removed server (i.e., the RMO will no longer communicate S ram directly (i.e., rather than connecting through a central 
with the IMO associated with the server). Servers may be server). In this alternative embodiment, the program 
added to the load-balancing scheme in a similar manner. 45 instances exchange information (e.g., via status objects 
In state 514, a specialized object is registered with the and /or other elements of a load-balancing framework) and 
load-balancing framework (e.g., created from ils object redirect chent requests as necessary to balance the requests 
class). In state 516, parameters concerning the operation of m a ccordance with the selected policy, 
the specialized object are set. Illustrative parameters include In another alternative embodiment of the invention, one 
an identity of the RMO, the frequency of information 50 or more elements of a load-balancing framework are corn- 
retrieval from the RMO, an identity of the lookup table, b «ied. By way of illustration, an RMO may be designed to 
method of interfacing with the RMO and/or lookup table, perform the functions of an IMO and collect information 
etc. In one embodiment of the invention, the specialized fr om one or more status objects, 
object analyzes the information collected from the servers What is claimed is: 

hosting the application instances, identifies a preferred 55 L A method of distributing requests for an application 

server in accordance with the load-balancing framework and among a plurality of application instances operating on a 

updates the lookup table. plurality of servers, wherein the requests are received at a 

Where, for example, the application comprises web central ^ melhod comprising: 
browsing on web servers, the specialized object may take the selecting a policy, said policy demonstratmg a first server- 
form of a DNS updater configured on a DNS server to 60 selection factor for selecting a preferred server to 
modify a DNS zone file to identify the server to which receive a request for the application; 
requests are to be routed. Similarly, where load balancing is executing a first status module to determine a first status 
being performed for an application operating in a master/ of said first server-selection factor for a first instance of 
slave relationship (e.g., a master process or server routes the application; 

requests to slave processes or servers), the specialized object 65 executing a second status module to determine a second 

updates a data structure or entry indicating a preferred status of said first server-selection factor for a second 

process or server. instance of the application; 
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receiving said first status at the central server; 
receiving said second status at the central server; 
examining said first status and said second status to select 

a preferred server; and 
storing an identifier of said preferred server on the central 

server; 

wherein said first server-selection factor comprises an 
application-specific detail. 

2. The method of claim 1, further comprising operating a 
server monitor module to receive said first status of said first 
server-selection factor from said first status module. 

3. The method of claim 2, wherein said operating a server 
monitor module comprises receiving a first status of a 
second server-selection factor from a third status module. 

4. The method of claim 2, wherein said server monitor 
module executes on said first server. 

5. The method of claim 2, wherein said server monitor 
module executes on the central server. 

6. The method of claim 1, further comprising executing a 
third status module to determine a first status of a second 
server-selection factor for said first instance of the applica- 
tion. 

7. The method of claim 6, wherein said first status module 
comprises said third status module. 

8. The method of claim 1, further comprising operating a 
central monitor module for receiving said first status and 
said second status. 

9. The method of claim 8, wherein said central monitor 
module executes on the central server. 

10. The method of claim 1, wherein said executing a first 
status module comprises operating a first status module 
residing on the first server. 

11. The method of claim 1, wherein said executing a first 
status module comprises communicating with a first instance 
of the application to determine a first status of said first 
server-selection factor. 

12. The method of claim 1, further comprising: 
selecting a local policy for a subset of the plurality of 

servers, said local policy specifying a local server- 
selection factor for selecting a server to receive a 
request for the application. 

13. The method of claim 1, wherein said application- 
specific detail comprises one of the set of: number of 
accesses to the application, number of requests for access to 
the application, number of electronic mail messages, size of 
electronic mailbox, number of electronic mailboxes. 

14. A method of load balancing requests for an application 
received at a central server among a set of servers, wherein 
each server in the set of servers operates' an instance of the 
application, comprising: 

selecting a policy for directing a request for the applica- 
tion to a preferred server, wherein said policy reflects a 
server factor for selecting said preferred server from the 
set of servers; 

configuring a first status object to determine a first status 
of said server factor for a first instance of the applica- 
tion; 

configuring a first server monitor object to receive said 
first status; 

configuring a central monitor object to receive multiple 
statuses of said server factor for multiple instances of 
the application, including said first status; 

examining said multiple statuses to select a preferred 
server; and 

updating the central server to identify said preferred 
server; 
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wherein said server factor comprises a performance indi- 
cator specific to the application. 

15. The method of claim 14, further comprising: 
executing said first status object, wherein said first status 

5 object resides on said first server; 

receiving said first status by said first server monitor 
object; and 

receiving said first status at the central server, by said 
central monitor object, from said first server monitor 
io object. 

16. The method of claim 15, wherein said executing said 
first status object comprises operating said first status object 
to periodically determine a status of said server factor for a 
first instance of the application. 

15 17. The method of claim 15, further comprising main- 
taining said first server monitor object on said first server. 

18. The method of claim 14, wherein the set of servers 
includes a subset, the method further comprising: 

configuring an intermediate central monitor object to 
20 collect one or more statuses of said server factor for one 

or more members of the subset; and 
receiving said one or more statuses at the central server 

from said intermediate central monitor object. 

19. The method of claim 18, further comprising selecting 
25 a local policy for balancing requests for the application 

among members of the subset according to a local server 
factor. 

20. The method of claim 19, wherein said local server 
policy is different from said policy. 

30 21. The method of claim 14, further comprising: 

executing said first status object, wherein said first status 

object resides on said central server; and 
maintaining said server monitor object on the central 

server. 

35 22. The method of claim 21, wherein said executing said 
first status object comprises operating said first status object 
to periodically determine a status of said server factor for a 
first instance of the application. 

23. The method of claim 14, wherein said central server 
40 comprises a lookup table to associate said preferred server 

with the application, and wherein said updating comprises 
storing an address of said preferred server. 

24. A method of distributing requests for an application 
among a plurality of application instances operating on a 

45 plurality of servers, the method comprising: 

selecting a policy for identifying a preferred server to 
receive a request for the application, said policy includ- 
ing a first server-selection factor; 
determining a status of said first server-selection factor for 

one or more instances of the application; 
storing an identifier of a preferred server on a central 
server; and 

directing a request for the application received after said 
55 storing to said preferred server; 

wherein said first server-selection factor comprises a 
performance indicator specific to the application. 

25. The method of claim 24, wherein said storing com- 
prises: 

60 examining said status of said first server-selection factor 
for said one or more instances of the application; and 
selecting a preferred server on the basis of said 
examination, said preferred server being associated 
with one of said one or more instances of the applica- 

65 tion. 

26. The method of claim 25, wherein said examining and 
said selecting are performed on said central server. 
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27. The method of claim 24, wherein said determining 
comprises executing a first status module to retrieve a first 
status of said first server-selection factor for a first instance 
of the application. 

28. The method of claim 27, wherein said first status 5 
module is executed on a server operating said first instance 
of the application. 

29. An apparatus for balancing requests for an application 
among multiple servers operating multiple instances of the 
application, wherein the requests are received at a central 10 
server, comprising: 

a first server for operating a first instance of the applica- 
tion; 

a second server for operating a second instance of the 
application; 15 

a first status module for determining a first application- 
specific status of said first instance; 

a second status module for determining a second 
application-specific status of said second instance; 20 

a first server monitor module for receiving said first 
application-specific status from said first status module; 

a second server monitor module for receiving said second 
application -specific status from said second status 
module; 25 

a central monitor module for receiving said first 
application-specific status and said second application - 
specific status; and 

an update module for updating the central server to 3Q 
indicate one of said first server and said second server 
to receive a request for the application. 

30. The apparatus of claim 29, wherein said first status 
module resides on said first server. 

31. The apparatus of claim 29, wherein said first status 35 
module determines said first application-specific status by 
receiving said first status from said first instance. 

32. The apparatus of claim 29, wherein said first server 
monitor module operates on said first server. 

33. The apparatus of claim 29, wherein said first server ^ 
monitor module operates on the central server. 

34. The apparatus of claim 29, wherein the central server 
comprises said central monitor module and said update 
module. 

35. The apparatus of claim 29, further comprising a server 45 
farm, said server farm comprising: 

one or more servers; and 

an intermediate central monitor module for receiving a 
status of an instance of the application operating on one 
of said one or more servers and communicating said 50 
status to said central monitor module. 

36. An apparatus for load balancing requests for an 
application received at a central server, comprising: 

a first status determination means for determining a first 
application-specific status of a first instance of the 55 
application; 
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a second status determination means for determining a 
second application -specific status of a second instance 
of the application; 

central monitor means for receiving said first application- 
specific status and said second application-specific sta- 
tus; 

server selection means for selecting a preferred server 
from one of said first server and said second server; and 

updating means for storing an identifier of said preferred 
server on the central server. 

37. The apparatus of claim 36, further comprising a first 
server monitor means for receiving said first application- 
specific status from said first status determination means. 

38. A computer readable storage medium storing instruc- 
tions that, when executed by a computer, cause the computer 
to perform a method for balancing* requests for an applica- 
tion among a plurality of servers, wherein the requests are 
received at a central server, the method comprising: 

selecting a policy for directing a request for the applica- 
tion to a preferred server, wherein said policy reflects a 
server factor for selecting said preferred server from the 
set of servers; 

configuring a first status object to determine a first status 
of said server factor for a first instance of the applica- 
tion; 

configuring a first server monitor object to receive said 
first status; 

configuring a central monitor object to receive multiple 
statuses of said server factor for multiple instances of 
the application, including said first status; 

examining said multiple statuses to select a preferred 
server; and 

updating the central server to identify said preferred 
server; 

wherein said first server-selection factor comprises an 
application-specific detail. 

39. A method of load-balancing multiple requests for an 
application, wherein instances of the application execute on 
a plurality of servers, the method comprising: 

receiving a client request for an application at a first server 
operating a first instance of the application; 

executing a first status module on the first server, wherein 
said first status module is configured to determine a first 
status of the first instance; 

executing a server monitor module on the first server, 
wherein said server monitor module is configured to 
receive a first status of a second instance of the appli- 
cation operating on a second server; 

examining said first status of the first instance and said 
first status of the second instance to select a preferred 
server from among the plurality of servers; and 

routing the client request to said preferred server. 
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[57] ABSTRACT 

In a telecommunications system containing a host computer 
and multiple real connections to the telecommunications 
network, an apparatus, method and system for allowing 
transmission to the host computer to reroute dynamically 
through the multiple available real connections to the host 
without requiring changes to the devices connecting to the 
host. 
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VIRTUAL INTERNET PROTOCOL (IP) 
ADDRESSING 

BACKGROUND OF THE INVENTION 

TCP/IP (Transmission Control Protocol/Internet Protocol) 
is the transport mechanism underlying the Internet. It is also 
the underlying protocol for many intranets and business 
applications in existence today. TCP/IP was started as an 
educational and scientific network. It was not designed to 
handle high-volume traffic with the requirement of avail- 
ability 7 days per week, 24 hours per day. TCP/IP was 
designed primarily as a fast transport mechanism. Because 
of this design point, there were few backup or redundancy 
measures incorporated into TCP/IP. 

Through the growth of the Internet, which includes the 
world wide web, requirements have arisen for higher avail- 
ability and greater reliability for host TCP/IP networks. This 
has become especially true where the TCP/IP host controls 
business applications or transactions. The design of TCP/IP 
is such that each physical network interface adapter has 
associated with it an address. This address is unique within 
the entire network and is the method by which all other 
devices communicate with the adapter or the devices con- 
nected through the adapter. If a given TCP/IP host has 
multiple interface adapters, the users communicating with 
the host must select an interface adapter which they chose to 
use. The user must then reference the host by the address of 
the particular adapter which the user has chosen to use. 

The above method works well when each host has one 
interface adapter or where the interface adapters never fail, 
but in large host systems where there are more than one 
interface adapter available, situations arise where one of the 
interface adapters fails. When this happens, under the cur- 
rent TCP/IP implementations, the information that is being 
sent to the failing interface adapter is incapable of being 
rerouted to the functioning interface adapters). The only 
method available to rectify the break in the communications 
link is to either replace the failing adapter and configure the 
new adapter with the same network address as the adapter 
that failed or to modify the application sending information 
to send information to the new network adapter, or to detect 
the failure, determine which resources are affected and 
broadcast the new location upon which the affected 
resources can be found. All of these potential solutions 
require the intervention of an operator or an applications 
programmer which will result in the loss of packets and the 
disruption of traffic. 

SUMMARY OF THE INVENTION 

The present invention allows for a dynamic rerouting of 
traffic from one network device to another available network 
device or adapter on the same host without the loss of 
packets or the intervention of an operator. This invention 
enhances the fault tolerance of a TCP/IP network using hosts 
with multiple or redundant devices or network interface 
adapters without significantly increasing the cost of the 
network. This is accomplished by the use of a virtual device, 
a virtual adapter and a virtual IP address (VIPA). The virtual 
device will be active as long as the host upon which it resides 
is active. The virtual adapter will have a home address of the 
virtual IP address, but there will be no physical interface 
directly associated with it. This allows traffic with the 
address of the virtual IP address to be routed through any of 
the available physical network interface devices that are 
running in the host utilizing traditional routing protocols 
such as Route Information Protocol (RIP). 
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In addition, broadcast or advertising techniques can be 
used to ensure that all the routers, hosts or other devices 
adjacent to the host which contains the virtual IP address are 
made aware of this virtual IP address such that they can use 
5 a highly available, highly reliable network interface address. 

DESCRIPTION OF THE DRAWINGS 

FIG. 1 depicts a portion of the prior art TCP/IP network. 
30 FIG. 2 depicts a portion of a TCP/IP network including 
the present invention. 

FIGS. 3A and 3B depicts the logic of notifying the 
network of the virtual IP address. 

FIG. 4 depicts the decision necessary for inserting a 
15 source address in the outgoing packet. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

The preferred embodiment of the present invention is 

20 implemented in, but not limited to, an IBM MVS host 
running the TCP/IP protocol. It allows for an IP address that 
selects a TCP/IP stack (and an MVS image if there is only 
one stack on the MVS image) without selecting a specific 
network device or attachment. Other hosts that connect to 

25 MVS TCP/IP applications can send data to the MVS virtual 
IP address via whatever paths are selected by the routing 
protocols. This provides additional tolerance of failures for 
devices and adapters attached to an MVS host. 

30 The means of accomplishing this task is to modify the 
configuration file of the host's TCP/IP software such that a 
single network interface device address is assigned to a host, 
this single address being a virtual address. This virtual 
address is then used as the address for notifying or indicating 

35 to other nodes where the particular host is located. In 
addition, the virtual IP address is used as the outgoing source 
address in the messages sent from the host to other nodes, 
hence the virtual IP address is the only interface address 
known for the system in the preferred embodiment. This 

4Q allows real, physical adapters to fail, be added or be removed 
without affecting the flow of messages. 

This flexibility is achieved by modifying the IP protocol 
software to treat such virtual devices and addresses differ- 
ently from physical devices. In particular, the software is 

45 modified to treat virtual devices as always being operational. 
Furthermore, the IP protocol software is modified to always 
use the virtual IP address for normal data traffic as the source 
IP address field of datagrams being sent out to the network. 
The IP software is also modified so that no start commands 

50 are necessary for virtual devices in the configuration file. 
Moreover, the routing software, also known as the RIP 
software or Route Information Protocol software, is modi- 
fied so that it need not receive RIP updates for the virtual 
routes. This reduces the amount of status traffic and allows 

55 the virtual interface and the virtual routes associated with it 
to remain "active" until an operator expressly deletes them. 
This is more fully described with reference to the figures. 

FIG. 1 depicts the prior art connection of a subset of a 
TCP/IP network. The portion of the network shown in FIG. 

60 1 contains two hosts, the MVS host 201 and the user's host 
210. The MVS host 201 is connected to the TCP/IP network 
225 by means of three different adapters. The first adapter, 
207, connects directly to the a router 205, through which it 
connects to the TCP/IP network 225. The second adapter 

65 208 and the third adapter 209 connect to router 216, which 
then connects through router 205 to the TCP/IP network and 
through router 215 into the TCP/IP network. This is intended 
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to display only a small subset of the available possibilities 
for connection into a TCP/IP network, it is not meant to limit 
the use of the invention in any way to subsets of the TCP/IP 
network. Without the present invention, if the host 210 
recognized the MVS host 201 as being at adapter 207, and 
adapter 207 failed, then all communications between the 
MVS host 201 and the host 210 would stop. 

FIG. 2 depicts a portion of a TCP/IP network in which the 
present invention is implemented. In the network of the 
present invention, a virtual adapter 250 is denned in the 
MVS host 201. The virtual adapter is defined to reference 
each of the real adapters available in the host (207, 208 & 
209) such that any other host sending messages to the MVS 
host need only reference the virtual adapter address and the 
messages will be send to the MVS host. This allows one or 
more of the adapters to fail or become unavailable without 
impacting the flow of traffic. This is achieved by advertising 
only the virtual route that represents the virtual IP address 
and allowing the normal routing protocols to route the 
information on the shortest available path. Continuing with 
the example from above, if adapter 207 fails, utilizing the 
present invention, the router 205 continues sending to the 
virtual adapter 230. When adapter 207 fails, the traffic is 
automatically rerouted through the network to adapter 208. 
If adapter 207, at a later time, becomes available then the 
traffic can be dynamically rerouted through adapter 207 
since, at that lime, the path through adapter 207 will be the 
shortest available path to the target virtual address. 

FIGS. 3 and 4 demonstrate the flow of information in the 
MVS host which permit this to happen. First, the virtual 
device and virtual link are defined by an operator. In the 
preferred embodiment this is done using the DEVICE and 
LINK statements in the network definition file. Once this 
happens the virtual routes representing the virtual IP address 
used to identify the host will be sent to all of the attached 
hosts and routers advertising the location of the MVS host. 
Once this is announced, if all packets targeted for the MVS 
host are assigned the virtual IP address as their target 
address, then the routing protocol will direct the packets to 
each of the virtual IP addresses via any of the active real 
adapters connected to the MVS host 201. 

In addition to the advertising of the routes to the virtual IP 
address, the internet protocol software of the MVS machine 
is modified to insert the virtual IP address into the source IP 
address field of messages which it transmits to other hosts. 
This ensures that the virtual IP address Ls the one that is used 
by other parties to the communication. The only exception 
to this is that the virtual address is not used in transmission 
of the RIP routing packets. The actual physical adapter 
address is used for the RIP routing packets as required by the 
routing protocol. 

FIG. 4 depicts the changes to the logical flow of creating 
the outgoing data block. As construction of the outgoing data 
block begins (405) a test is done to determine if the the 
source address has already been defined (410). This will be 
true when the data block came from another source outside 
the local host. If the source address is not yet defined a test 
is done to determine whether source virtual IP addressing is 
enabled (415). This is done by testing the option named 
SOURCEVIPA. If SOURCEVIPA is enabled, then a test is 
made to determine whether or not SOURCEVIPA should be 
ignored (420). This would be the case when RouteD was 
being used for RIP packets. In general, the IGNORE 
SOURCEVIPA option is not set for TCP, UDP (except in the 
case of RIP) and ICMP datagrams. If the IGNORE 
SOURCEVIPA is enabled, then the physical address is 
inserted into the outgoing data block (435). If the IGNORE 
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SOURCEVIPA is not enabled, the virtual IP address is 
inserted into the outgoing data block (425). 

Returning to block 415, if the source address was not 
defined (410) and the SOURCEVIPA option was not 
enabled, then the physical address is inserted into the 
outgoing data block (435). All cases then return to a com- 
mon logic flow and send the outgoing data block over the 
physical interface (430). 

FIG. 3 depicts the changes to the logical flow of process- 
ing the expiration of the broadcast timer in the MVS host of 
the preferred embodiment. Beginning with FIG. 3 A, in the 
preferred embodiment, the first step of the logic is to 
initialize or scan when the interface timer pops (301). Next 
the list of interfaces is read into memory (303). A test is 
made to determine whether the interface is new or has been 
reactivated (305). If the interface is not new or reactivated, 
then a test is made to determine if the interface is a virtual 
interface (307). If the interface is not a virtual interface, the 
status of the interface is obtained (309) and a test is made to 
determine whether this real interface is active (311). If the 
interface is a real interface and it is not active the routes for 
the interface are marked for timeout and deletion (313). A 
test is then made to determine whether the entire list of 
interfaces has been processed (315). 

At the test to determine whether the interface was a virtual 
interface (307), had the interface been a virtual one, the logic 
flow would have gone directly to the test for the end of list 
(315) since virtual interfaces are never inactivated by tim- 
eouts in the preferred embodiment. If, when a test was made 
as to whether the interface were new or reactivated (305), it 
was determined that the device were new or reactivated, a 
test would then be made to determine whether the interface 
were a virtual interface (317). If it were a virtual interface, 
the virtual IP address net, subnetwork and host routes would 
be added to the daemon routing table (319). If it were 
determined that the new or reactivated interface were not a 
virtual interface (317), then the real net subnetwork and host 
routes for the interface would be added to the routing table 
(321). At this point all of the logic flows converge on a test 
to determine whether the end of the interface list has been 
reached (315). If the end of the list has not been reached, 
then flow returns to block 303 to continue reading the list. 
If the end of the list has been reached, the logic flow 
continues on to FIG. 3B. 

FIG. 3B begins with the timer popping or expiring (330) 
which causes the application to loop through the interfaces 
in the interface list (332). A test is made to determine 
whether that next interface is a virtual interface (334). If it 
is a virtual interface, a test is made to determine whether the 
end of the interface list has been reached (340), if the end of 
the this has not been reached then the pointer to the next 
interface is incremented (332). If the next retrieved interface 
is not a virtual interface (334) then a test is made to 
determine if the interface is active (336). If the real interface 
is not active, a test is made to determine whether the end of 
the list has been reached (340) and if the end of the interface 
list has not been reached then the logic flow returns to 
retrieving the next item in the interface list (332). If the real 
interface is active (336), then virtual and non-virtual routes 
are advertised using the daemon routing table. These are sent 
over the physical interfaces. At this point a test is made to 
determine whether the end of the interface list has been 
reached (340). If the end of the interface list has not been 
reached, control returns to 332. 

If at any point it was determined that the end of the 
interface list had been reached (340), the process would then 
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receive the routing updates (342) and make a test to deter- 
mine whether there was a virtual route match (344). If there 
is a virtual route match, the process would continue (350). 
If it were determined that there was not a virtual route match, 
a test is made to determine if a route exists that is shorter 5 
(346) based on whatever metric is chosen for the system. If 
it were determined that a shorter route does exist, then the 
route is replaced (348) by the shorter route. In either case, 
the process continues on its normal flow. 
What is claimed is: 10 

1. A host computer connected to a communications 
network, said host containing two or more network interface 
adapters, each of said network interface adapters having a 
unique physical address assigned, said network interface 
adapters connecting said host computer to said communi- 15 
cations network wherein information may be dynamically 
rerouted through any of said two or more network interface 
adapters onto said communications network, said dynamic 
rerouting comprising the steps of: 

creating a virtual IP interface adapter for said host 20 
wherein said virtual IP interface adapter is recognized 
as being continuously operational; 

allocating a virtual IP address for said virtual IP interface 
adapter; and, ^ 

replacing each of said physical addresses of said network 
interface adapter with said virtual IP address in all 
routing advertisements utilizing existing routing pro- 
tocols to dynamically reroute said information. 

2. A host computer as claimed in claim 1 wherein each of 3Q 
said physical addresses is replaced with said virtual IP 
address in outgoing data blocks. 

3. A host computer as claimed in either of claims 1 or 2 
wherein the communications protocol being used is TCP. 

4. A host computer as claimed in either of claims 1 or 2 35 
wherein the communications protocol being used is Internet 
Control Message Protocol (ICMP). 

5. A host computer as claimed in either of claims 1 or 2 
wherein the communications protocol being used is User 
Datagram Protocol (UDP) but not Routing Information ^ 
Protocol (RIP). 

6. A host computer as claimed in either of claims 1 or 2 
wherein two or more virtual IP addresses are assigned to said 
host. 

7. A method for connecting host computer to a commu- 45 
nications network, said host containing two or more network 
interface adapters, each of said network interface adapters 
having a unique physical address assigned, said network 
interface adapters connecting said host computer to said 
communications network wherein information may be 5Q 
dynamically rerouted through any of said two or more 
network interface adapters onto said communications 
network, said dynamic rerouting comprising: 

means for creating a virtual IP interface adapter for said 
host; 55 

means for allocating a virtual IP address for said virtual IP 
interface adapter wherein said virtual IP interface 
adapters are recognized as being continuously opera- 
tional; and, 
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means for replacing each of said physical addresses of 
said network interface adapter with said virtual IP 
address in all routing advertisements utilizing existing 
routing protocols to dynamically reroute said informa- 
tion. 

8. A method as claimed in claim 7 wherein each of said 
physical addresses is replaced with said virtual IP address in 
outgoing data blocks. 

9. A method as claimed in either of claims 7 or 8 wherein 
the communications protocol being used is TCP. 

10. A method as claimed in either of claims 7 or 8 wherein 
the communications protocol being used is Internet Control 
Message Protocol (ICMP). 

11. A method as claimed in either of claims 7 or 8 wherein 
the communications protocol being used is User Datagram 
Protocol (UDP) and not Routing Information Protocol 
(RIP). 

12. A method as claimed in either of claims 7 or 8 wherein 
two or more virtual IP addresses are assigned to said host. 

13. A media containing programmable code for execution 
on a host computer connected to a communications network, 
said host containing two or more network interface adapters, 
each of said network interface adapters having a unique 
physical address assigned, said network interface adapters 
connecting said host computer to said communications 
network wherein information may be dynamically rerouted 
through any of said two or more network interface adapters 
onto said communications network, said dynamic rerouting 
comprising the steps of: 

programmable means for creating a virtual IP interface 
adapter for said host; 

programmable means for allocating a virtual IP address 
for said virtual IP interface adapter wherein said virtual 
IP interface adapters are recognized as being continu- 
ously operational; and, 

programmable means for replacing each of said physical 
addresses of said network interface adapter with said 
virtual IP address in all routing advertisements utilizing 
existing routing protocols to dynamically reroute said 
information. 

14. A media containing programmable code as claimed in 
claim 13 wherein each of said physical addresses is pro- 
grammably replaced with said virtual IP address in outgoing 
data blocks. 

15. A media containing programmable code as claimed in 
cither of claims 13 or 14 wherein the communications 
protocol being used is TCP. 

16. A media containing programmable code as claimed in 
either of claims 13 or 14 wherein the communications 
protocol being used is Internet Control Message Protocol 
(ICMP). 

17. A media containing programmable code as claimed in 
either of claims 13 or 14 wherein the communications 
protocol being used is User Datagram Protocol (UDP) and 
not Routing Information Protocol (RIP). 

18. A media containing programmable code as claimed in 
either of claims 13 or 14 wherein two or more virtual IP 
addresses are assigned to said host. 
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